Skip to content

bundle camx & iris-vpu & audioreach DMKS modules - #80

Closed
Christopher Obbard (obbardc) wants to merge 111 commits into
qcom/debian/latestfrom
wip/obbardc/more-dkms
Closed

bundle camx & iris-vpu & audioreach DMKS modules#80
Christopher Obbard (obbardc) wants to merge 111 commits into
qcom/debian/latestfrom
wip/obbardc/more-dkms

Conversation

@obbardc

@obbardc Christopher Obbard (obbardc) commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Tested with local kernel build with additional DKMS modules.

Will NOT build on resolute, due to camx-dkms (and others) not available.
Will NOT build on debian, due to audioreach-dkms not available in all branches.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
…merges in a dedicated step

Previously, only one qcom-next PR could be specified per input. With this update:

Multiple PR support

- Accept space-separated PR numbers in the qcom-next-pr input (e.g., 12 45 78).
- Iterate through each PR, fetch its head, and merge sequentially.
- Abort and fail the job immediately if any merge results in a conflict.

Empty input handling

If no PRs are provided (empty or default value), the step is skipped gracefully without error.

Separate workflow step

Moves qcom-next PR merge logic out of the sync kernel step into its own dedicated step for better clarity and maintainability.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Implement dynamic logic to construct build_info file containing key environment variables and kernel build configuration parameters:

JOB_ID
JOB_ATTEMPT
ORG_NAME
REPO_NAME
QCOM-BUILD-UTILS BRANCH/TAG
QCOM-BUILD-UTILS HEAD SHA
KERNEL BRANCH/TAG
KERNEL HEAD SHA
PRs FROM QCOM-NEXT
PRs FROM KERNEL TOPICS

To be uploaded alongside kernel deb package in the S3 bucket.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Added a temporary cron schedule for testing purposes.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
- Add a README section describing the build-kernel CI workflow, including:
  - Manual (workflow_dispatch) vs nightly scheduled execution at 05:00 UTC / 21:00 PST
  - Supported inputs, their defaults, and how they influence the build
  - High-level kernel build and Debian packaging pipeline using qcom-build-utils (kernel sync, optional PR integration, config enablement, kernel build, deb packaging)
  - Provenance tracking via build_info metadata for each run

- Introduce a mermaid diagram illustrating the workflow from triggers
  through inputs to final outputs, to provide an at-a-glance system view

- Clarify final S3 artifact layout, including bucket name and run-specific path convention (ORG_NAME/pkg/temp/REPO_NAME/RUN_ID-RUN_ATTEMPT), to improve discoverability and traceability of produced kernel .deb packages


Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Added note about planned migration to native Debian tooling for kernel build orchestration and package generation.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Add a DTB manifest file defining the ordered set of device-tree blobs to be included in the combined DTB image for Qualcomm platforms.

The manifest currently covers:
- QCM6490 IDP
- QCS6490 RB3 Gen2
- Lemans EVK
- QCS9100 Ride (R0/R3)
- QCS8300 Ride
- Monaco EVK
- QCS615 Ride
- SA8775P Ride (R0/R3)
- Hamoa IoT EVK

This file is consumed as a configuration parameter by the combined DTB image tooling (e.g. build-dtb-image.sh in the image build pipeline) and provides a single source of truth for the DTB set and ordering used when generating dtb.bin for UEFI-based boot flows and CI builds.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Extend the pkg-linux-kernel build-kernel CI workflow to generate and publish a combined DTB image (dtb.bin) alongside the kernel Debian artifacts.

Changes:
- After the kernel build completes, invoke qcom-build-utils/kernel/scripts/build-dtb-image.sh inside the kmake-image:ubuntu-noble-arm64 container to produce dtb.bin:
  -dtb-src   = qcom-build utils/kernel/out/arch/arm64/boot/dts/qcom
  -manifest  = pkg-linux-kernel/config/dtb-manifest
  -size      = 4
  -out       = dtb.bin
- Add an in-container size sanity check that:
  - Iterates all DTB entries from the manifest (skipping blank/comment lines) under the DTB source tree.
  - Sums their byte sizes and compares the total against the size of dtb.bin.
  - Fails the job if the sizes do not match, catching manifest/DTB mismatches early in CI.
- Refactor artifact assembly into a dedicated host-side step for
  readability:
  - Create qcom-build-utils/kernel/deb_artifact/.
  - Copy all generated kernel .deb packages into deb_artifact/.
  - Copy dtb.bin into deb_artifact/ when present to publish the combined DTB image as a first-class CI artifact.
- Keep the existing build_info generation and S3 upload logic unchanged, so the final artifact set uploaded under: ${ORG_NAME}/pkg/temp/${REPO_NAME}/${JOB_ID} ${JOB_ATTEMPT}/ now includes kernel .debs, dtb.bin, and build_info in a single deb_artifact/ directory.

This makes the combined DTB image part of the standard kernel build pipeline and ensures that the dtb.bin produced in CI is both manifest-driven and size-validated.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Update the build-kernel workflow to source DTBs from the in-tree kernel build output instead of the out/ scratch directory.

Specifically:
- Change DTB_SRC from:
    $BUILD_TOP/out/arch/arm64/boot/dts/qcom
  to:
    $BUILD_TOP/qcom-next/arch/arm64/boot/dts/qcom

This aligns the combined DTB image generation with the actual location of the built DTBs produced by the qcom-next kernel build.


Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Temporarily disable the combined DTB image (dtb.bin) build step for `schedule`-triggered runs in the build-kernel workflow by guarding the step with `if: ${{ github.event_name != 'schedule' }}`.

Rationale:
- The dtb.bin generation relies on build-dtb-image.sh and related DTB tooling that currently exists only on a feature branch of qcom-build-utils.
- Scheduled jobs consume the default `main` for qcom-build-utils, which does not yet contain the DTB image tooling, causing the scheduled pipeline to fail when attempting to build dtb.bin.

With this change:
- `workflow_dispatch` runs (where we can explicitly select a branch of qcom-build-utils that includes the DTB tooling) still build and publish dtb.bin alongside kernel .deb artifacts.
- Nightly `schedule` runs continue to exercise the kernel build and packaging paths without being blocked by the in-flight combined-DTB tooling work.

This guard is intended as a temporary measure until the DTB image build logic is merged into the qcom-build-utils mainline and becomes available to scheduled CI runs.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
…ed to qcom-distro-images workflow

Revert the last six commits to remove the combined DTB image build from the
kernel repository. DTB image generation is now handled by the qcom-distro-images
workflow.

Reverted range: HEAD~6..HEAD
Reverted commits:
- 85851ca ci: gate combined DTB image build on non-scheduled runs
- dfa70e1 Update dtb-manifest to temporarily include single dtb only
- 3d86510 ci: fix DTB source path for combined DTB image build
- 5c48c7a Remove combined dtb size sanity check
- da87d29 ci: integrate combined DTB image build into kernel workflow
- b1a366f config: add dtb-manifest for combined DTB image generation
Extends the `build-kernel` workflow to accept an optional user-defined `kernel-url`. This enhancement allows the pipeline to ingest and build kernel sources from other repositories outside of the standard `qualcomm-linux` GitHub hierarchy.

Key changes:
- Introduced `kernel-url` input to the workflow dispatch trigger.
- Implemented mutually exclusive sync logic:
  - **Default:** Retains existing behavior (syncs from `qualcomm-linux/kernel`, applies PR merges/patches) when no URL is provided.
  - **Custom:** Clones directly from the provided `kernel-url` and explicitly skips PR merge and patch application steps to ensure source integrity.
- Refactored the local workspace directory structure, renaming `qcom-next` to `qcom-kernel` to decouple the build environment from specific branch naming conventions.
- Updated all downstream path references and config scripts to align with the new workspace layout.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Updates the `Generate build_info metadata` step to include the `CUSTOM KERNEL URL` field. This ensures that the source origin of the build artifacts is explicitly traceable for auditing and debugging purposes, particularly when non-default repositories are used.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
Adds a workflow_dispatch-only workflow for testing the debian kernel
packaging work on feat/kernel-debian-source-packaging before it merges.

To use: Actions → build-kernel-deb → Run workflow → select branch
feat/kernel-debian-source-packaging.

Signed-off-by: Adam Bickett <abickett@qti.qualcomm.com>
Signed-off-by: Adam Bickett <abickett@qti.qualcomm.com>
- Add input box for pkg-linux-qcom PR and workflow step to handle application of self-PRs
- Add workflow step to extract REPO and ORG name
- Switch to using docker-pkg-build main repo/branch
- Add workflow step to upload to s3 bucket location

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
…nner

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
…ch or commit

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
… steps

Restructure the build-kernel-deb workflow so that kernel source setup is
handled by explicit workflow steps rather than being delegated to
build-kernel.sh. The script is now invoked with --local-source, receiving
a fully prepared kernel tree from the workflow.

New steps added:

  Sync kernel source
    Resolves the target ref (latest qcom-next-* tag via git ls-remote, or
    the kernel-branch input as-is), performs a shallow clone into
    $GITHUB_WORKSPACE/kernel-source, and exports KERNEL_DIR, KERNEL_REF,
    and KERNEL_SHA to the job environment for use by downstream steps.

  Merge qcom-next PR patches  (conditional: qcom-next-pr != '')
    For each space-separated PR number, fetches pull/<N>/head from the
    kernel remote and merges it with --no-commit. Merge conflicts abort
    the build immediately. Already-merged or fast-forwarded PRs emit a
    warning and are skipped cleanly.

  Apply kernel-topics PR patches  (conditional: kernel-topics-pr != '')
    For each space-separated PR number, downloads the GitHub-generated
    .patch file from qualcomm-linux/kernel-topics and applies it with
    git am, preserving authorship and commit messages. Patch failures
    abort the build immediately.

The Build kernel package step is updated to pass --local-source instead
of --repo/--branch/--tag/--latest-tag. build-kernel.sh skips all git
operations and proceeds directly to debian/ injection, prepare, and build.

build-kernel.sh is not modified. Its clone/fetch capability is preserved
for developer use; CI simply uses the --local-source entry point.

Two new workflow_dispatch inputs are added:
  qcom-next-pr      space-separated PR numbers from qualcomm-linux/kernel
  kernel-topics-pr  space-separated PR numbers from qualcomm-linux/kernel-topics

Both default to empty and are no-ops when not supplied, preserving full
backward compatibility with existing runs.

Signed-off-by: Bjordis Collaku <bcollaku@qti.qualcomm.com>
@obbardc Christopher Obbard (obbardc) changed the title bundle camx & iris-vpu DMKS modules bundle camx & iris-vpu & audioreach DMKS modules Sep 1, 2026
@obbardc

Copy link
Copy Markdown
Contributor Author

Seems like PR builds don't include "qli-staging" repo in the deps. Will resolve in separate PR.

The packaging (debian/, prepare-source.sh, build-kernel.sh) and the CI
generator (.github/workflows/, ci/) have lived on two branches whose
histories only meet at the initial commit, so every build had to check
the two out separately and keep their assumptions in sync by hand.

This split has worked out in practice to not be ideal: changes which
touch both the packaging and the CI generator will need two separate PRs
which have to be kept in sync. Instead of this extra complication, merge
the packaging and CI branches so that PRs may target just one branch
and have the ability to modify both packaging and CI.

Merge the packaging branch into the CI branch to keep the 107 commits
of packaging history.

Two files conflict during the merge:

  - README.md: The two branches hold different documents rather than
    two versions of the same file; so the packaging README.md is moved to
    debian/README.md and the top-level README keeps the generator focus.
    Links between the two files follows in a separate commit; any
    improvements/deduplication will follow in separate PRs.

  - .github/workflows/qcom-preflight-checks.yml. Deleted from the main
    branch in 18b0905 and modified since on the packaging branch; the
    deletion stands for now.

CONTRIBUTING.md and SECURITY.md merge to the versions on the main
branch: the packaging branch never touched them after the fork point,
so the <REPLACE-ME> placeholders 9af567c filled in stay filled in.

pr-build.yml comes across as the packaging branch left it, still pinned
to main and to a separate ci-ref. Rewiring it (and the rest of the
cross-branch plumbing) follows in a separate commit.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
The merge left two READMEs describing two halves of one repository with
no way to get from either to the other and debian/README.md still
opening as though it were a repository of its own.

Point each at the other and reword them slightly to describe their
intention. No content moves between them yet as they are different
documents.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
With debian/ and ci/ now on one branch, the cross-branch plumbing that
kept them together has nothing left to do. Every build leg took its
packaging from qcom/debian/latest and its CI scripts from a separate
ci-ref, which meant a matrix change and the packaging change it depended
on could land in either order and only meet at build time.

Drop pkg_linux_qcom_ref from the matrix and both the pkg-linux-qcom-ref
and ci-ref inputs from build-kernel-deb.yml and build-kernel-ubuntu.yml,
and collapse the prepare job's two checkouts into one. That checkout now
carries no ref at all, leaving actions/checkout on github.sha: the commit
the run was dispatched from, or the one the calling workflow runs at.
No ref override is left, so a build is always the packaging and the CI
of a single commit.

Drop self-pr with it. It applied a pkg-linux-qcom PR on top of the
packaging checkout, which is what pr-build.yml now does for every PR by
construction. Pointing a dispatched build at a PR is worth having back
as a first-class input, but not as a knob that fetches over the ref the
rest of the run is pinned to; leave it for a follow-up.

The CI scripts are 100755 in the tree, so dropping the copy step's
chmod loses nothing.

pr-build.yml arrived with the merge as the packaging branch left it and
is adapted to the same model: it triggers on PRs to main, calls
./.github/workflows/build-kernel-deb.yml rather than pinning the
reusable workflow to @main, and resolves the matrix from the PR head
instead of main. A PR is now tested entirely against its own commit.
Checking main out for the matrix no longer isolated anything, since the
build legs run the PR's ci/ scripts either way.

Daily and Release revisions are unchanged; resolve-matrix.sh emits the
same debian_revision for all nine legs.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
…-main

Merge packaging branch into the CI branch
…tely

BUILD_EXCLUSIVE_CONFIG may name several kernel options, space separated, all
of which must be set for dkms to build the module. The gate report tested the
whole string as a single symbol, which never matches, so every gate was
reported as unset:

    | CONFIG_ARCH_QCOM CONFIG_PM_DEVFREQ CONFIG_SYNC_FILE is NOT set

That is worse than saying nothing: it accuses the kernel config whatever the
real cause was, and it did exactly that while the actual failure was a broken
source tree. Both kgsl and iris-vpu declare multi-option gates, so the report
was wrong for every module that reached it.

Evaluate each option separately.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Add camx to debian/dkms-modules so the Qualcomm camx driver is built
against the kernel being packaged and shipped inside
linux-image-<KVER>-qcom and declare camx-dkms in Build-Depends.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Add iris-vpu to debian/dkms-modules so the Qualcomm video accelerator
driver is built against the kernel being packaged and shipped inside
linux-image-<KVER>-qcom and declare iris-vpu-dkms in Build-Depends.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Add audioreach to debian/dkms-modules so the audioreach driver is built
against the kernel being packaged and shipped inside
linux-image-<KVER>-qcom and declare audioreach-dkms in Build-Depends.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Comment on lines +204 to +208
- name: Checkout pkg-linux-qcom
# debian/ and ci/ live on the same branch, so one checkout supplies
# both the packaging and resolve-kernel-ref.sh, derive-localversion.sh,
# derive-debian-revision.sh and the suite_suffix_mapping.
uses: actions/checkout@v4
Comment on lines +210 to +215
- name: Checkout docker-pkg-build
uses: actions/checkout@v4
with:
repository: qualcomm-linux/docker-pkg-build
ref: main
path: docker-pkg-build
Comment on lines +80 to +88
- name: Checkout debusine-action helpers
uses: actions/checkout@v5
with:
repository: qualcomm-linux/debusine-action
ref: ${{ env.DEBUSINE_ACTION_REF }}
path: debusine-action
fetch-depth: 1
sparse-checkout: |
lib
Comment on lines +150 to +158
- name: Checkout debusine-action helpers
uses: actions/checkout@v5
with:
repository: qualcomm-linux/debusine-action
ref: ${{ env.DEBUSINE_ACTION_REF }}
path: debusine-action
fetch-depth: 1
sparse-checkout: |
lib
Comment on lines +42 to +46
- name: Checkout pkg-linux-qcom
uses: actions/checkout@v4

# Pinned to @main while native kernel-build support is still landing
# upstream in qualcomm-linux/docker-pkg-build.
Comment on lines +47 to +52
- name: Checkout docker-pkg-build
uses: actions/checkout@v4
with:
repository: qualcomm-linux/docker-pkg-build
ref: main
path: docker-pkg-build
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v4
Comment on lines +59 to +66
- name: Checkout qcom-build-utils
uses: actions/checkout@v4
with:
repository: qualcomm-linux/qcom-build-utils
ref: ${{ inputs.qcom-build-utils-ref }}
token: ${{ secrets.DEB_PKG_BOT_CI_TOKEN }}
path: ./qcom-build-utils
fetch-depth: 1
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants